The present invention relates to test executive software for organizing and executing test sequences to control instrumentation systems.
A test executive is a program that allows a user to organize and execute sequences of reusable test modules to control a test involving one or more instruments. The test modules often have a standard interface and typically can be created in a variety of programming environments. The test executive software operates as the control center for the automated test system. More specifically, the test executive software allows the user to create, configure, and/or control test sequence execution for various test applications, such as production and manufacturing test applications. Text executive software typically includes various features, such as test sequencing based on pass/fail results, logging of test results, and report generation, among others.
Test executives include various general concepts. The following comprises a glossary of test executive nomenclature.
Code Modulexe2x80x94A program module, such as a Windows Dynamic Link Library (.dll) or LabVIEW VI (.vi), that contains one or more functions that perform a specific test or other action.
Test Modulexe2x80x94A code module that performs a test.
Stepxe2x80x94Any action, such as calling a test module to perform a specific test, that the user can include within a sequence of other actions.
Step Modulexe2x80x94The code module that a step calls.
Sequencexe2x80x94A series of steps that the user specifies for execution in a particular order. Whether and when a step is executed can depend on the results of previous steps.
Subsequencexe2x80x94A sequence that another sequence calls. The user specifies a subsequence call as a step in the calling sequence.
Sequence Filexe2x80x94A file that contains the definition of one or more sequences.
Sequence Editorxe2x80x94A program that provides a graphical user interface for creating, editing, and debugging sequences.
Run-time Operator Interfacexe2x80x94A program that provides a graphical user interface for executing sequences on a production station. A sequence editor and run-time operator interface can be separate application programs or different aspects of the same program.
Test Executive Enginexe2x80x94A module or set of modules that provide an API for creating, editing, executing, and debugging sequences. A sequence editor or run-time execution operator interface uses the services of a test executive engine.
Application Development Environment (ADE)xe2x80x94A programming environment such as LabVIEW, LabWindows/CVI, or Microsoft Visual C, in which the user can create test modules and run-time operator interfaces.
Unit Under Test (UUT)xe2x80x94The device or component that is being tested.
Prior art test executive programs are generally hard-coded programs with little flexibility, modularity or configurability. Prior art systems generally include executive code, sequence code, test modules, and then instrument drivers. The executive code typically includes functionality such as UUT identification, operator notification, test report generation, and logging results. The vendor may or may not provide source code for the executive code to the end-user. If the vendor did provide source code for the executive code, the above-mentioned functionality would be subsumed in that source code but it would not be in a format which is easily configurable. In other words, this functionality was not separated out as a single configurable entity. Therefore, if the user desired to modify this functionality, the user would have to hunt through the source code to find and edit the code, which is a tedious process. Thus, in prior art, the user would be required to enter the programming language to change this functionality. This was tedious and difficult.
Therefore, an improved test executive program is desired which provides improved flexibility, modularity and configurability.
Prior art test executive programs also do not easily allow user configuration of steps. In general, step functionality was hard-coded with little flexibility, modularity or configurability. Therefore, an improved test executive program is also desired which provides improved configurability of steps.
An improved test executive program is further desired which provides improved type conflict resolution and improved automatic result collection.
The present invention comprises an improved test executive program which provides process model and step type functionality for improved flexibility, modularity and configurability. The test executive program of the present invention also provides distributed type storage and conflict resolution, as well as automatic result collection.
The test executive of the present invention, referred to as the TestStand test executive software, is a flexible, powerful test executive framework which provides a number of novel features. In particular, the test executive software of the present invention provides greatly improved configurability and modularity, thus simplifying the creation, modification and execution of test sequences. The test executive software provides numerous ways for the user to modify the default configuration and components or to add new components. These extensibility mechanisms enable the user to create the test executive that meets the user""s particular requirements without modifying the TestStand test execution engine.
The test executive software expands on the traditional test executive concepts and introduces many new ones. The test executive software of the present invention includes new concepts and features, including step types, step properties, sequence variables, sequence parameters, module adapters, and process models.
The TestStand test executive software architecture includes operator interface programs for interfacing to various software programs. The TestStand test executive software also includes a sequence editor for editing sequences. The sequence editor and the operator interface programs interface to the test executive engine, referred to as the TestStand Engine. One or more process models couple to the TestStand Engine. The TestStand Engine interfaces through an adapter interface to one or more adapters. The adapters in turn interface to code modules and sequence files.
The TestStand Engine plays a pivotal role in the TestStand architecture. The TestStand Engine executes sequences, wherein sequences contain steps that can call external code modules. By using module adapters that have the standard adapter interface, the TestStand Engine can load and execute different types of code modules. TestStand sequences can call sub-sequences through the common adapter interface. TestStand uses a special type of sequence called a process model to direct the high-level sequence flow. The TestStand Engine exports an ActiveX Automation API used by the TestStand sequence editor and run-time operator interfaces.
Typesxe2x80x94Conflict Resolution
The TestStand test executive system of the present invention includes various types, including step types, custom named data types, and standard named data types. A type can thus be a data type or step type. For each type that a file uses, the TestStand system stores the definition of the type in the file. More specifically, when a user creates a type in the test executive system, the user creates the type using a graphical user interface (GUI) and assigns a name to the type. The user may also assign version data to the type and/or the system may apply a timestamp or other version data to the type. When the user later stores a step or data of the at least one type in a file, the TestStand Engine automatically stores a type definition of the type in the file in response. The user can also specify that a file always saves the definition for a type, even if the file does not currently use the type. Because many files can use the same type, many files can contain definitions for the same type.
The TestStand system allows only one definition for each type to exist in memory. If the user modifies the type in one view, the type is updated in all views. However, problems can arise if there is not a single definition of a type in the system. For example, when the user creates a step type, and then the user stores an instance of the step type in the sequence file, the type definition of the step type is stored in the file. If someone later changes the contents of the step type, and then reloads the sequence which is based on the original step type, there will be a conflict. This conflict can also occur when the user moves a sequence file from a first computer to a second computer, wherein the second computer includes a different type definition for the respective type name.
When the user loads a file which includes a stored type, the method of the present invention executes to perform conflict resolution. Here presume that the user loads a file with at least one type wherein the file preferably stores a type definition of the loaded type, e.g., a data type or step type. In response to the user loading the file, the TestStand Engine automatically determines the type being loaded, and then automatically determines if the loaded type conflicts with one or more previously loaded/registered types. This determination includes comparing the type being loaded with the previously loaded/registered types in the system. In the preferred embodiment, the determination of a type conflict comprises determining if the name of the loaded type conflicts with the name of any of the previously loaded/registered types.
If the types do not conflict, then the TestStand Engine registers the type in the system. Thus, since there is no type conflict, the type can be safely registered in the system. If the types do conflict, then the TestStand Engine selects a type in response to determining the type conflict, preferably using the version data associated with each type. In the preferred embodiment, the TestStand Engine selects the type with the version data indicating the most recent version, e.g., the one with the most recent version number.
In one embodiment, the TestStand Engine requests user input regarding selection of a type in response to determining a type conflict, and this user input is used in selecting the type. In other words, when the TestStand Engine detects a type conflict, the Engine displays a dialog informing the user of the conflict and requesting the user to decide which type should be used. The dialog also preferably displays the version data from each of the types to aid the user in the selection. The user then selects the type to be used.
After selection of the type, the TestStand Engine conforms existing instances of one or more non-selected types with the selected type. This conforming comprises modifying the data structure of the one or more non-selected types to conform to the data structure of the selected type. This conforming also includes preserving at least a portion of the old data of the one or more non-selected types.
The conflict resolution performed according to the present invention is especially important, for example, where the user creates and stores the type in a first computer, and then later loads the file in a second computer. For example, the second computer may include a different type which includes the same name as the type created and stored in the first computer, thereby creating a conflict.
Step Types
A sequence comprises a series of steps, wherein a step is typically a test performed on an instrument. The present invention includes step types which allow more modular and configurable steps while minimizing user coding and effort.
In a test sequence with a number of steps, in many instances the user will desire a number of steps that have some commonality of functionality and/or properties. A primary purpose of a step type is to define common properties and/or operations associated with a plurality of steps in a single location, referred to as the step type, thereby eliminating the need for the user to define these common properties and/or operations with each of the respective steps. Instances of the step type incorporate this common functionality and/or properties from the step type, and thus the user is not required to hard code that functionality and/or properties in each instance or step. The functionality and/or properties defined by a step type is generally peripheral or associated with the actual test or the step being performed.
A step type thus essentially comprises a custom set of properties and/or operations associated with a step. Stated another way, a step type defines common operations and/or data that are associated with a test module in a similar way that a process model defines functionality associated with calling the main sequence file of a test. The test module is hard coded, and the step type represents operations and/or properties associated with calling this test module. A step type is a modular, identifiable unit configured by the user, preferably through a graphical user interface, e.g., dialogs.
Step types define common functionality by defining an edit sub-step and pre and post sub-steps. The pre and post sub-steps execute before and after, respectively, a step. The edit sub-step is used to configure the peculiarities of a particular instance of a step type. For instance the edit sub-step can be configured to display a message or dialog to request user. This message is typically displayed at configuration time, not run time. Step types also include pre and post expressions and status expressions for configurability. These expressions are executed at run time.
A step type has similar functionality to a type definition, meaning that once the user has configured a step type and used it throughout different steps of the sequence, if the user later changes that step type, those changes propagate through all of the steps which are based on that step type. This type definition functionality is not performed through copying or propagating changes. Rather, the imformation is held within the step type, and during run time the sequence examines the step type to determine the step type parameters for execution of the step.
In prior art test executives, the functionality performed by the step type of the present invention was hard coded in the test executive itself and was not easily changeable. Further, this functionality was not in a modular form which could be reused or applied to a plurality of steps. The step type of the present invention embodies this functionality, e.g., the pre and post operations in a typical test, and places this in a configurable and reusable form. Step types of the present invention are modular and user configurable and provide tremendous savings in developer effort.
Step types are generally configured and used as follows. First, the user configures a first step type. As discussed above, the first step type defines common functionality and common data for steps (instances) of the first step type. The common functionality preferably includes one or more of a common editing interface, pre-step functionality and post-step functionality. The common functionality is preferably specified by the user as one or more sub-steps that call code modules written in any of various programming languages. The common data includes one or more property values organized in a user-defined hierarchy. Configuration of the first step type also optionally includes creating one or more code templates, wherein the code templates are useable in creating a code module called by instances of the first step type.
The user then creates a test sequence file for testing the unit under test. The test sequence file includes at least one sequence comprising a plurality of steps, wherein one or more of the steps are of the first step type (instances of the first step type).
The test sequence file is then executed to test the unit under test. Execution of the test sequence file includes executing each of the plurality of steps in the at least one sequence. When steps of the first step type are executed, execution includes executing the common functionality for the one or more of the steps which are of the first step type and also includes utilizing the common data for the one or more of the steps which are of the first step type. More specifically, for a step of the first step type, i.e., an instance of the first step type, execution of the step includes:
executing the pre-step functionality;
executing a code module referenced by the step after executing the pre-step functionality; and
executing the post-step functionality after executing the code module.
Process Models
The present invention includes process models which provide a modular and configurable entity for encapsulating operations and functionality associated with a class of test sequences. The process model thus encapsulates a xe2x80x9ctesting processxe2x80x9d for a plurality of test sequences. The xe2x80x9ctesting processxe2x80x9d includes common operations required to be performed before and after the test executive executes the sequence that performs the tests. Common operations include identifying the UUT, notifying the operator of pass/fail status, generating a test report, and logging results.
The process model enables the user to write different test sequences without repeating standard testing operations in each sequence. The process model is also user-modifiable. This is important because the testing process can vary based on the production line, the production site, or the systems and practices of the company in which the test executive is used. The process model thus provides tremendous savings in user development time.
TestStand provides a mechanism for defining a process model. A process model is in the form of a sequence file. The user can edit a process model just as he/she edits other sequences. TestStand ships with a fully functional default process model. The user can write his/her own process model, or the user can copy the default process model and then modify it.
Process models are used in the creation of a test system as follows. First the user configures a process model, or alternatively selects a previously configured process model. The process model includes functionality which is common to a plurality of different test sequences, wherein the functionality is for testing one or more UUTs. As discussed above, the process model is a separate user configurable module, e.g., is modularly separate from the test sequence file. More specifically, the process model is modularly separate from the test executive engine, the operator interfaces, the test sequences, and the test code. The process model preferably comprises a sequence file and is user editable as a sequence file. The process model comprises a plurality of steps which call code modules, wherein the code modules can be written in a variety of different programming languages. The process model includes one or more of pre-test operations, post-test operations, variables, parameters, types and code modules. The process model is also independent of language and construction of the operator interface program. Finally, the user can configure the process model to provide one or more of: customized report generation, database logging, UUT identification, and UUT status notification, among others.
Configuration of the process model includes one or more of: configuring pre-test operations and/or post-test operations in the process model; configuring one or more entry points in the process model; changing the order of calls to callback functions; adding/deleting calls to callback functions; adding new callback functions in response to new calls added to the process model; overriding callback functions called by the process model; modifying existing callback functions in the process model; and configuring variables, parameters, types, steps and/or code modules.
The user then creates a test sequence file, also called a client file, for testing the unit under test. The test sequence file includes the test sequence for the desired UUT, which is called MainSequence. The MainSequence file is called by the process model file and typically includes a number of sub-sequences which call into other sequence files. The user may include one or more client callbacks in the test sequence file which operate to override or replace callbacks in the process model.
The process model is comprised in a process model sequence file. The process model sequence file is a file that has multiple sequences that can be of types entry point sequence, callback sequence or normal sequence. A normal sequence is typically a utility subsequence. The entry point sequence is a place where the user can begin execution within the process model sequence file. The callback sequence is a sequence which typically performs test process functionality and is called by a call in the process model. The callback sequence in the process model sequence file is the default callback for the call, and is replaceable or overrideable by callback sequences inserted into the client file.
The process model and the test sequence file are then executed to test the unit under test. In the preferred embodiment, the user invokes execution of the process model, and during execution of the process model the process model operates to call the test sequence file. The user preferably begins execution of the process model through a graphical user interface, such as through a menu. The user preferably selects a desired execution entry point to begin execution within the process model sequence file. The test sequence file or client file can operate with any of the execution entry points, i.e., the test sequence file is not customized for a particular entry point in the process model. This way, for a single client file, the user can choose different ways of running the respective client file.
According to the present invention, the operator interface program automatically updates in response to user configuration of one or more entry points. When the user defines one or more entry points in the process model, the operator interface program queries the test executive engine to determine the one or more entry points defined by the user in the process model. The operator interface program then displays the one or more entry points in response to the query. Thus the one or more entry points configured by the user in the process model automatically appear in the operator interface program. Stated another way, changes to the process model which affect the appearance of the operator interface do not require manual changes to the operator interface, but rather the operator interface automatically updates itself.
Automatic Result Collection
The TestStand test executive system of the present invention performs automatic result collection to automatically collect the results of each step. The user can enable or disable result collection for a particular sequence or for the entire test station.
In the preferred embodiment, each sequence has a local array that stores the results of each step. The contents in the results for each can vary depending on the step type. When TestStand stores the results for a step into the array, it adds information such as the name of the step and its position in the sequence. For a step that calls a sequence, TestStand also adds the result array from the subsequence.
The automatic result collection of the present invention preferably operates as follows. First, the user creates a test sequence file for testing the unit under test, wherein the test sequence file includes at least one sequence comprising a plurality of steps. The user then configures the results to be collected. This, for example, includes defining one or more properties to be collected for one or more of the steps in the sequence. Where one or more step types are included in the test sequence file, the user preferably configures results for the step types.
The user then enables automatic result collection. In one embodiment, automatic result collection may be the default result collection technique. Also, the user can enable or disable automatic result collection for particular sequence(s) or for the entire test station. The test sequence file is then executed to test the unit under test. Execution of the test sequence file includes executing each of the plurality of steps in the sequence.
The TestStand Engine operates to automatically collect the results of each step in the sequence during execution. If the user enabled automatic result collection, then the automatic collection is performed in response to this enabling, and for the specified sequences. During execution, the user can also dynamically configure the results to be collected, as desired.
The automatic collection includes collecting standard result information, wherein the standard result information is automatically collected regardless of the user input configuring the results to be collected. The standard result information includes one or more of: a name of the step and a position of the step in the sequence.
Where the test sequence file includes a step type and the user has configured results for the step type, the method automatically collects results depending on the step type of the steps in the sequence. For example, when the steps in the test sequence file include a first step having a first step type and a second step having a second step type, the automatic collection operates to collect first results for the first step according to the first step type and operates to collect second results for the second step according to the second step type.
In the preferred embodiment, the sequence includes a local array that stores the results of each step in the sequence, and the automatic collection includes storing the results of each step in the local array of the sequence. The local array is preferably a local variable of the sequence, wherein the local array is accessible from within the sequence and from other code modules.
When a step in a sequence of the test sequence file calls a sub-sequence, the automatic collection includes storing a result array from the sub-sequence as the result for the first step. Further, when a first step calls a sub-sequence, and the sub-sequence includes a step which in turn calls another-subsequence, the result for the first step includes a chronology and hierarchy of step execution made in response to the first step.
Where the first step calls a sub-sequence, and the sub-sequence executes asynchronously, the automatic collection includes allocating an array element for the result when the first step calls the sub-sequence and storing the result in the array element after completion of the sub-sequence.